Abstract: In recent years, China's coal mining is huge, and the healthy operation of the mining equipment gear transmission system is a prerequisite for ensuring efficient coal mining and safe underground operation. Gear transmission device as the core component of mechanical equipment, its operating conditions are complex, and the vibration signals of many structural components overlap, making it more difficult to monitor and requiring in-depth analysis by professionals. In view of the challenges faced by the monitoring of the gear transmission system of mining machinery and equipment, this paper firstly reduces the vibration signals based on the joint noise reduction algorithm of modal preference reconstruction and expression (CEEMDAN), and further constructs a fault diagnosis model based on the long and short-term memory (LSTM) neural network. At the same time, the gray wolf algorithm is improved in three aspects, namely, population initialization method, convergence parameter updating strategy, and location updating strategy, and after performance testing and comparison with other algorithms, it is found that the three improvement measures adopted in this paper significantly enhance the performance of the gray wolf algorithm. Finally, the improved Gray Wolf algorithm is used to optimize the hyperparameters of the fault diagnosis model. After simulation analysis and experimental verification, the fault diagnosis model proposed in this paper, compared with the traditional model, the diagnostic accuracy and stability have been greatly improved, and the average accuracy of identification can reach more than 96%.